Systems and methods for remapping three-dimensional gestures onto a finite-size two-dimensional surface

ABSTRACT

A method for operating a real-time gesture based interactive system includes: obtaining a sequence of frames of data from an acquisition system; comparing successive frames of the data for portions that change between frames; determining whether any of the portions that changed are part of an interaction medium detected in the sequence of frames of data; defining a 3D interaction zone relative to an initial position of the part of the interaction medium detected in the sequence of frames of data; tracking a movement of the interaction medium to generate a plurality of 3D positions of the interaction medium; detecting movement of the interaction medium from inside to outside the 3D interaction zone at a boundary 3D position; shifting the 3D interaction zone relative to the boundary 3D position; computing a plurality of 2D positions based on the 3D positions; and supplying the 2D positions to control an application.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit of U.S. Provisional Patent No.61/988,845, titled “Systems and Methods for Remapping Three-DimensionalGestures onto a Finite-Size Two-Dimensional Surface,” filed in theUnited States Patent and Trademark Office on May 5, 2014, the entiredisclosure of which is incorporated herein by reference.

BACKGROUND

Aspects of embodiments of the present invention relate to systems andmethods for capturing three dimensional gestures and mapping thecaptured gestures onto a user interface for a finite-sizedtwo-dimensional surface. Such systems and methods may be implemented bycomputing devices including mobile phones, tablets, desktop and laptopcomputers, etc.

Three dimensional gesture-based user interface systems may use depth mapacquisition systems to detect a gesture made by a user. These gesturesmay be detected without the user's making physical contact with aportion of the device (e.g., without touching a screen or pressing abutton).

Computational and energy efficiency of such gesture-based user interfacesystems may be considerations in processing and power constrainedenvironments such as mobile devices. Furthermore, higher computationalefficiency may result in better user experiences due to reduced lag andbetter responsiveness.

SUMMARY

Aspects of embodiments of the present invention are directed to systemsand methods for providing a real-time gesture based user interfacesystem. Some embodiments of the present invention are directed tosystems and methods for improving computational efficiency of detectinggestures. Some embodiments are directed to systems and methods forproviding user friendly interfaces.

According to one embodiment, a real-time gesture based interactivesystem includes: a processor; an acquisition system configured tocapture a sequence of frames of data for constructing a depth map of afield of view of the acquisition system; memory storing an interactionmedium tracking application, the interaction medium tracking applicationconfiguring the processor to: obtain frames of data from the acquisitionsystem; compare successive frames of the frames of data for portionsthat change from one frame to the next; determine whether any of theportions that changed are part of an interaction medium detected in thesequence of frames of data; define an inner 3D interaction zone relativeto an initial position of the part of the interaction medium detected inthe sequence of frames of data, where the inner 3D interaction zonecorresponds to a bounded region that is less than the frame of data andthat contains the part of the interaction medium detected in thesequence of frames of data; track a movement of the interaction mediumin the data to generate a plurality of 3D positions of the interactionmedium; detect movement of the interaction medium from inside to outsidethe inner 3D interaction zone at a boundary 3D position; shift the inner3D interaction zone relative to the boundary 3D position; compute aplurality of 2D positions based on the 3D positions; and supply the 2Dpositions to control an application.

The acquisition system may include at least one of: a plurality ofcameras in a stereo arrangement having overlapping fields of view; aninfrared camera; a color visible light camera; and an illuminationsource configured to generate at least one of visible light, infraredlight, ultrasonic waves, and electromagnetic signals.

The memory may further store instructions of the interaction mediumtracking application to configure the processor to: compute a 3Dvelocity of the interaction medium within the inner 3D interaction zonebased on the 3D positions; and compute two-dimensional movement datacorresponding to the 3D positions and the 3D velocity, differences inthe two-dimensional movement data being non-linear with respect todifferences in the 3D positions.

The interaction medium may be a portion of a human body.

The memory may further store instructions of the interaction mediumtracking application to configure the processor to shift the inner 3Dinteraction zone relative to the boundary 3D position by computing aconvex combination of the boundary 3D position and the center of theinner 3D interaction zone.

The memory may further store instructions of the interaction mediumtracking application to configure the processor to detect the movementof the interaction medium from the inside to the outside of the inner 3Dinteraction zone based on the 3D positions.

The memory may further store instructions of the interaction mediumtracking application to configure the processor to detect the movementof the interaction medium from the inside to the outside of the inner 3Dinteraction zone by detecting a coarse movement of the interactionmedium within an entire portion of the field of view of the acquisitionsystem.

The memory may further store instructions of the interaction mediumtracking application to configure the processor to: detect adisengagement event; and stop tracking the movement of the interactionmedium in response to the disengagement event.

The memory may further store instructions of the interaction mediumtracking application to configure the processor to detect thedisengagement event by: comparing the interaction medium detected in theframes of data to a target configuration to generate a compatibilityconfidence level; and detecting the disengagement event when theconfidence level is below a threshold level.

The memory may further store instructions of the interaction mediumtracking application to configure the processor to detect thedisengagement event by: comparing the interaction medium detected in theframes of data to a disengagement configuration; and detecting thedisengagement event when the disengagement configuration is detected.

The memory may further store instructions of the interaction mediumtracking application to configure the processor to: define an outer 3Dinteraction zone surrounding the inner 3D interaction zone; and detectthe disengagement event by detecting a movement of the interactionmedium from inside to outside of both the inner 3D interaction zone andthe outer 3D interaction zone within a disengagement time period.

The memory may further store instructions of the interaction mediumtracking application to configure the processor to: track the movementof the interaction medium along a first direction toward a boundary ofthe inner interaction zone and concurrently update the plurality of 2Dpositions in accordance with the movement along the first direction;shift the inner interaction zone along the first direction andconcurrently stop updating the plurality of 2D positions in accordancewith movement along the first direction; and track the movement of theinteraction medium along a second direction opposite the first directionand concurrently start updating the plurality of 2D positions inaccordance with the movement along the second direction.

According to one embodiment of the present invention, a real-timegesture based interactive system includes: a processor; an acquisitionsystem configured to capture a sequence of frames of data forconstructing a depth map of a field of view of the acquisition system;memory storing an interaction medium tracking application, theinteraction medium tracking application configuring the processor to:obtain frames of data from the acquisition system; compare successiveframes of the of frames of data for portions that change from one frameto the next; determine whether any of the portions that changed are partof an interaction medium detected in the sequence of frames of data;define an inner 3D interaction zone relative to an initial position ofthe part of the interaction medium detected in the sequence of frames ofdata, where the inner 3D interaction zone corresponds to a boundedregion that is less than the frame of data and that contains the part ofthe interaction medium detected in the sequence of frames of data; tracka movement of the interaction medium in the data to generate a pluralityof 3D positions of the interaction medium; compute a 3D velocity of theinteraction medium within the inner 3D interaction zone based on the 3Dpositions; compute a plurality of 2D positions based on the 3D positionsand the 3D velocity, differences in the 2D positions being non-linearwith respect to differences in the 3D positions; and supply the 2Dpositions to control an application.

The 3D velocity may include a horizontal component v_(x) and a verticalcomponent v_(y) and wherein the 3D velocity is thresholded by a minimumthreshold T_(m) and a maximum threshold T_(M) to compute thresholded 3Dvelocities a and b, where:

a=min(max(|v _(x) ^(t) |,T _(m)),T _(M)),

b=min(max(|v _(y) ^(t) |,T _(m)),T _(M)).

The real-time gesture based interactive system may further include adisplay device having S_(w) columns and S_(h) rows, wherein theplurality of 2D positions includes column data c and row data rcorresponding to coordinates of the display device, and wherein the 3Dvelocity and the thresholded 3D velocities are mapped onto the columndata c and the row data r, where:

c _(t)=min(max(c _(t-1) −v _(x) ^(t) *K*a,0),S _(w))

r _(t)=min(max(r _(t-1) −v _(y) ^(t) *K*b,0),S _(h)).

and where K is a sensitivity parameter.

The 3D velocity may be an average of a plurality of velocities computedduring a time window.

According to one embodiment of the present invention, a method foroperating a real-time gesture based interactive system includes:obtaining a sequence of frames of data from an acquisition systemconfigured to capture data for constructing a depth map of a field ofview of the acquisition system; comparing, by a processor, successiveframes of the frames of data for portions that change from one frame tothe next; determining, by the processor, whether any of the portionsthat changed are part of an interaction medium detected in the sequenceof frames of data; defining, by the processor, an inner 3D interactionzone relative to an initial position of the part of the interactionmedium detected in the sequence of frames of data, where the inner 3Dinteraction zone corresponds to a bounded region that is less than theframe of data and that contains the part of the interaction mediumdetected in the sequence of frames of data; tracking, by the processor,a movement of the interaction medium in the data to generate a pluralityof 3D positions of the interaction medium; detecting, by the processor,movement of the interaction medium from inside to outside the inner 3Dinteraction zone at a boundary 3D position; shifting, by the processor,the inner 3D interaction zone relative to the boundary 3D position;computing, by the processor, a plurality of 2D positions based on the 3Dpositions; and supplying the 2D positions to control an application.

The acquisition system may include at least one of: a plurality ofcameras in a stereo arrangement having overlapping fields of view; aninfrared camera; a color visible light camera; and an illuminationsource configured to generate at least one of visible light, infraredlight, ultrasonic waves, and electromagnetic signals.

The method may further include: computing a 3D velocity of theinteraction medium within the inner 3D interaction zone based on the 3Dpositions; and computing two-dimensional movement data corresponding tothe 3D positions and the 3D velocity, differences in the two-dimensionalmovement data being non-linear with respect to differences in the 3Dpositions.

The interaction medium may be a portion of a human body.

The method may further include: shifting the inner 3D interaction zonerelative to the boundary 3D position by computing a convex combinationof the boundary 3D position and the center of the inner 3D interactionzone.

The method may further include detecting the movement of the interactionmedium from the inside to the outside of the inner 3D interaction zonebased on the 3D positions.

The may further include detecting the movement of the interaction mediumfrom the inside to the outside of the inner 3D interaction zone bydetecting a coarse movement of the interaction medium within an entireportion of the field of view of the acquisition system.

The method may further include: detecting a disengagement event; andstopping tracking the movement of the interaction medium in response tothe disengagement event.

The detecting the disengagement event may include: comparing theinteraction medium detected in the frames of data to a targetconfiguration to generate a compatibility confidence level; anddetecting the disengagement event when the confidence level is below athreshold level.

The detecting the disengagement event may include: comparing theinteraction medium detected in the frames of data to a disengagementconfiguration; and detecting the disengagement event when thedisengagement configuration is detected.

The method may further include: defining an outer 3D interaction zonesurrounding the inner 3D interaction zone; and detecting thedisengagement event by detecting a movement of the interaction mediumfrom inside to outside of both the inner 3D interaction zone and theouter 3D interaction zone within a disengagement time period.

The may further include: tracking the movement of the interaction mediumalong a first direction toward a boundary of the inner interaction zoneand concurrently update the plurality of 2D positions in accordance withthe movement along the first direction; shifting the inner interactionzone along the first direction and concurrently stop updating theplurality of 2D positions in accordance with movement along the firstdirection; and tracking the movement of the interaction medium along asecond direction opposite the first direction and concurrently startupdating the plurality of 2D positions in accordance with the movementalong the second direction.

According to one embodiment of the present invention, a method fortracking a gesture includes: obtaining a sequence of frames of data froman acquisition system configured to capture data for constructing adepth map of a field of view of the acquisition system; comparing, by aprocessor, successive frames of the frames of data for portions thatchange from one frame to the next; determining, by the processor,whether any of the portions that changed are part of an interactionmedium detected in the sequence of frames of data; defining, by theprocessor, an inner 3D interaction zone relative to an initial positionof the part of the interaction medium detected in the sequence of framesof data, where the inner 3D interaction zone corresponds to a boundedregion that is less than the frame of data and that contains the part ofthe interaction medium detected in the sequence of frames of data;tracking, by the processor, a movement of the interaction medium in thedata to generate a plurality of 3D positions of the interaction medium;computing, by the processor, a 3D velocity of the interaction mediumwithin the inner 3D interaction zone based on the 3D positions;computing, by the processor, a plurality of 2D positions based on the 3Dpositions and the 3D velocity, differences in the 2D positions beingnon-linear with respect to differences in the 3D positions; andsupplying, by the processor, the 2D positions to control an application.

The 3D velocity may include a horizontal component v_(x) and a verticalcomponent v_(y) and wherein the 3D velocity is thresholded by a minimumthreshold T_(m) and a maximum threshold T_(M) to compute thresholded 3Dvelocities a and b, where:

a=min(max(|v _(x) ^(t) |,T _(m)),T _(M))

b=min(max(|v _(y) ^(t) |,T _(m)),T _(M)).

The application may include a user interface displayed on a displaydevice having S_(w) columns and S_(h) rows, wherein the plurality of 2Dpositions comprises column data c and row data r corresponding tocoordinates of the display device, and wherein the 3D velocity and thethresholded 3D velocities are mapped onto the column data c and the rowdata r, where:

c _(t)=min(max(c _(t-1) −v _(x) ^(t) *K*a,0),S _(w))

r _(t)=min(max(r _(t-1) −v _(y) ^(t) *K*b,0),S _(h)).

and where K is a sensitivity parameter.

The 3D velocity may be an average of a plurality of velocities computedduring a time window.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, together with the specification, illustrateexemplary embodiments of the present invention, and, together with thedescription, serve to explain the principles of the present invention.

FIG. 1 is a block diagram of a gesture based interactive systemaccording to one embodiment of the present invention.

FIG. 2 is a block diagram of an image processing system according to oneembodiment of the present invention.

FIG. 3 is a conceptual illustration of a template that can be used toperform template matching of human fingers according to one embodimentof the present invention.

FIG. 4 is a flowchart illustrating a method for calculating a 2D cursorposition from a 3D movement detected based on 3D data from anacquisition system.

FIGS. 5A and 5B are conceptual illustrations of inner and outerinteraction zones according to one embodiment of the present invention.

FIG. 6 is a flowchart illustrating a method for shifting an interactionzone according to one embodiment of the present invention.

FIGS. 7A, 7B, and 7C are flowcharts illustrating methods for detecting adisengagement event according to some embodiments of the presentinvention.

FIG. 8 is a flowchart illustrating a method for repositioning aninteraction zone according to one embodiment of the present invention.

DETAILED DESCRIPTION

In the following detailed description, only certain exemplaryembodiments of the present invention are shown and described, by way ofillustration. As those skilled in the art would recognize, the inventionmay be embodied in many different forms and should not be construed asbeing limited to the embodiments set forth herein. Like referencenumerals designate like elements throughout the specification.

Some aspects of embodiments of the present invention are directed tosystems and methods for providing a user interface with motion dependentinputs. According to some aspects, embodiments of the present inventionallow a user to interact with a program, such as an operating system, awindow manager, or a video game, by making gestures in front of anacquisition device (e.g., a camera) of a computing device such as amobile phone, tablet computer, game console, or laptop computer. Theuser may form the gestures using body parts such as hands or fingers orby using a tool such as a stylus or wand. The computing device may usecomputer vision techniques to analyze data captured by the acquisitionsystem (e.g., video data captured by the camera) to detect the gesturesmade by the user. Such gestures may be made without the user's makingphysical contact with the computing device with the gesturing part ofthe body (e.g., without pressing a button or touching a touch sensitivepanel overlaid on a display). The object (e.g., the body part or tool)detected and tracked by the computing device to recognize the gestureswill be referred to herein as the “interaction medium.”

Examples of systems and methods for capturing and tracking gestures madeby a user are disclosed in U.S. Pat. No. 8,615,108 “Systems and Methodsfor Initializing Motion Tracking of Human Hands,” filed in the UnitedStates Patent and Trademark Office on Jul. 22, 2013, issued on Dec. 24,2013, the entire disclosure of which is incorporated herein by referenceand in U.S. Pat. No. 8,655,021 “Systems and Methods for Tracking HumanHands by Performing Parts Based Template Matching Using Images fromMultiple Viewpoints,” filed in the United States Patent and TrademarkOffice on Jul. 15, 2013, issued on Feb. 18, 2014, the entire disclosureof which is incorporated herein by reference.

Aspects of embodiments of the present invention are directed to systemsand methods for improving user interaction experiences in gesturetracking systems. One aspect of embodiments of the present inventionrelates to a non-linear mapping between a detected gesture and themovement supplied to a user interface. For example, a cursor displayedin a user interface may move a distance that corresponds both to thedistance moved by a detected finger and the speed with which the fingermoved during the gesture.

Another aspect of embodiments of the present invention relates toreducing computational load by performing gesture detection only withinan interaction zone smaller than an entire field of view of a gesturedetection system and moving the location of the interaction zone as theuser's interaction medium (e.g., a hand or a stylus) moves over time.Another aspect of embodiments of the present invention relates tomethods for allowing a user to move an interaction zone without causingchanges in the user interface.

Still another aspect of embodiments of the present invention relates toa disengagement gesture for allowing a user to disengage gesturetracking of the interaction medium.

FIG. 1 is a block diagram of a gesture based interactive systemaccording to one embodiment of the present invention. Referring to FIG.1, the gesture-based interactive system 10 includes an image processingsystem 12 configured to receive data captured by an acquisition system15 which may include at least one camera 14, 16. In some embodiments,the gesture based interactive system 10 processes the captured data todetermine the location and/or pose (e.g., orientation and shape) of agesture made by a user, where the gesture may be performed by aninteraction medium such as a body part (such as a hand, a finger, or anarm) or a control device (such as a stylus or wand). These gestures aregenerally performed at a distance from the gesture based interactionsystem (e.g., in the air and more than 6 inches from the acquisitionsystem) with no physical contact between the interaction medium and thegesture based interactive system 10 (e.g., without contact with a touchsensitive panel). The image processing system 12 may be further coupledto a display device 18 for displaying data relating to an application(e.g., a drawing application) running on the image processing system 12.

FIG. 2 is a block diagram of an image processing system according to oneembodiment of the present invention. The image processing system 12includes a processor 22 that is configured to communicate with anacquisition system interface (or camera interface) 24 coupled to theacquisition system 15 and a display interface 26 coupled to the displaydevice 18. The image processing system 12 also includes a memory 28,which can take the form of one or more types of storage includingsemiconductor and/or disk based storage. In the embodiment shown in FIG.2, the processor 22 may be configured using an operating system 30stored in the memory, but embodiments of the present invention are notlimited thereto and may be utilized without an operating system. Thememory 28 may also store a hand tracking application 32 (or interactionmedium tracking application) and an interactive application 34. When theuser executes hand gestures within the field of view of the acquisitionsystem 15, the hand tracking application 32 processes data received viathe acquisition system interface 24 (e.g., video image data fromcameras) to track the interaction medium and to detect gestures. Thedata may be processed by the processor 22 to detect gestures and motion.The detected gestures and motion may then be supplied as user input data(or movement data) to the operating system 30 and/or the interactiveapplication 34.

In various embodiments of the present invention, the image processingsystem 12 may be implemented in a variety of forms of hardware andsoftware including, but not limited to, a general purpose processorcoupled to dynamic and static memory, a graphics processing unit (GPU),a field programmable gate array (FPGA), an application specificintegrated circuit (FPGA), and combinations thereof.

The acquisition system interface 24 may be an appropriate interface forconnecting to peripheral devices such as a universal serial bus (USB)interface. The display interface 26 may be an interface for driving adisplay device such as a DVI interface, an HDMI interface, or aDisplayPort interface.

In some embodiments of the present invention, gestures made by a trackedinteraction medium are used to control motion within an application 34with respect to a 2D surface such as a display device 18 coupled to theimage processing system 12 by remapping detected changes in 3D positionof the interaction medium to changes in position on the 2D surface. Forexample, gestures may be used to control the movement of a cursor on thedisplay device 18 to allow the selection of user interface elements onthe display device. As another example, in a drawing application thecursor may also be used to identify the location of a drawing tool on avirtual canvas. As still another example, the gestures may be used tocontrol the direction in which virtual player is looking in a threedimensional video game (e.g., controlling the look direction in a firstperson game). However, for the sake of convenience, these user inputswill be described herein as corresponding to the movement of a cursor onscreen.

The interaction medium may be tracked using, for example, a templatematching process utilizing a template similar to the template shown inFIG. 3. The illustrated template 80 defines an edge 82 and a pluralityof pixel locations 84. The edge 82 can be utilized to perform edgedetection using techniques including, but not limited to, the imagegradient orientation technique disclosed in Hinterstoisser et al.,“Gradient Response Maps for Real-Time Detection of Texture-Less Objects”IEEE Transactions on Pattern Analysis and Machine Intelligence (2012),the disclosure of which is incorporated by reference herein in itsentirety. The pixel locations 84 can be utilized to identify surfacepixels on a candidate feature for the purpose of performing skin colorclassification. Although a specific template is illustrated in FIG. 4,any of a variety of templates can be used to identify parts of a humanhand in various configurations including, but not limited to, fingers,hands with one or more pointing fingers, closed and open hands, as wellas other interaction media such as styluses and wands.

Generally, detecting gestures is performed within an interaction zonethat is smaller than an entire space in which the acquisition system iscapable of detecting gestures. For example, the interaction zone mayonly include a small portion within an entire field of view covered bythe acquisition system 15.

In one embodiment of the present invention, a user may perform a “wakeup gesture” using the interaction medium at a 3D position P_(w) withinthe field of view of the acquisition system 15, where:

P _(w) =[x _(w) ,y _(w) ,z _(w)]^(T)

The interaction zone may then be initially defined as a cuboid centeredat P_(w) and having dimensions Δ_(w)=[Δ_(w) ^(x),Δ_(w) ^(y),Δ_(w)^(z)]^(T). The movement of the interaction medium within the interactionzone generates 3D positions P^(t) over time, where P^(t)=[x^(t), y^(t),z^(t)]^(T), t=0, 1, . . . , T where the coordinates may correspond to,for example, the center of a detected fingertip, the center of a hand,or the tip of a stylus.

Points within the interaction zone may be remapped onto the displaydevice 18 according to:

$\begin{bmatrix}c \\r\end{bmatrix} = {\begin{bmatrix}\frac{\Delta_{w}^{c}}{\Delta_{w}^{x}} & 0 \\0 & \frac{\Delta_{w}^{r}}{\Delta_{w}^{y}}\end{bmatrix}\begin{bmatrix}{x - x_{w}} \\{y - y_{w}}\end{bmatrix}}$

where c and r are the column and row of the remapped point within thereference frame of the display device 18, Δ_(w) ^(c) and Δ_(w) ^(r) arethe number of columns and the number of rows on the screen, x and y arethe current position of interaction medium along the x and y axes (e.g.,parallel to the plane substantially equidistant from the acquisitiondevice 15), and the point with coordinates [r C]^(T)=[0 0]^(T) is thecoordinate at center of the display device 18. The rows and columnsΔ_(w) ^(c) and Δ_(w) ^(r) of the screen may correspond to the number ofpixels in the width S_(w) and height S_(h) of the screen, butembodiments of the present invention are not limited thereto (e.g., ascaling factor may be applied between the rows and the height andbetween the columns and the width).

This methodology may be referred to as “linear” or “absolute” remapping,and allows the remapping of a 3D position in space to the screen in amanner that is substantially independent of the distance (along the zaxis) from the acquisition system. This allows for a consistent userexperience (e.g., the same size gestures may be used across thefunctional range of the acquisition system), and also allows the zcomponent of the detected motion to be used for other functionalities.This also allows movement of the cursor to any point in the displaydevice 18 by moving the user's body part or interaction device to acorresponding point in the interaction zone.

By limiting gesture detection to the interaction zone, computationalload is reduced because only a limited region is analyzed for gestures.In addition, the detection of false positives, such as undesired movingobjects outside of the interaction zone, is reduced.

On the other hand, using a linear remapping as described above requiresthe user to limit his or her interactions to gestures within theinteraction zone. The location of the interaction zone may be fixedbased on the wakeup position P_(w). Moving the tracked hand out of theinteraction zone may cause undesired effects. In addition, a user maylose track of the size or extent of the interaction zone and maytherefore unintentionally exit the interaction zone during normal usage,which may frustrate the user. Further, non-linear acceleration, such asthat typically used for computer mice and trackpads, is not easilyimplemented with a fixed interaction zone because such interactionsgenerally result in a drifting effect. For example, when using a fixedinteraction zone, if a user's hand repeatedly moves quickly to the leftand slowly to the right, then after a few iterations it may no longer bepossible to move a cursor to the right edge of the display.

As such, aspects of embodiments of the present invention are directed tomoving the interaction zone in accordance with the movement of a user'sbody part (e.g., hand) or control device (e.g., stylus) whilemaintaining performance benefits of a fixed size interaction zone andwhile providing consistent mapping across the field of view of theacquisition system.

Non-Linear Interactions

As discussed above, in a linear remapping between three dimensional (3D)gestures and a two dimensional (2D) user interface, various points in 3Dspace are mapped directly to corresponding points in the 2D userinterface. FIG. 4 is a flowchart illustrating a method for calculating a2D cursor position from a 3D position detected based on 3D data from anacquisition system.

In one embodiment, assuming that the display device 18 is a display witha width of S_(w) pixels and a height of S_(h) pixels, and assuming thata wake up event occurs at time t=0, the coordinates of the interactionmedia are initialized to be [x⁰, y⁰, z⁰]^(T) and a cursor havingcoordinates (c, r) is initially manned to the center of the screen:

$\left\lbrack {{c^{0} = \frac{S_{w}}{2}},{r^{0} = \frac{S_{h}}{2}}} \right\rbrack^{T}$

At each successive frame (t>0), the image processing system 12 receivesdata from the acquisition system 15 (e.g., video frame data from stereocameras). The hand tracking application 32 then detects 3D movement inthe received acquisition system data and generates positions P^(t),where P^(t)=[x^(t), y^(t), z^(t)]^(T), t=0, 1, . . . , T in operation402. The 3D position data is then used in operation 404 to compute the3D velocity v^(t) of the gesture at time t as:

[v _(x) ^(t) ,v _(y) ^(t) ,v _(z) ^(t)]^(T) =[x _(t) −x _(t-1) ,y _(t)−y _(t-1) ,z _(t) −z _(t-1)]^(T)

In some embodiments of the present invention, only the x and ycomponents of the gestures (e.g., in a plane of points that aresubstantially equidistant from the acquisition system) are consideredfor remapping purposes, while the z component (e.g., along the axis ofthe acquisition system) may be used for other interactions such aszooming.

In addition, in some embodiments of the present invention, themagnitudes of the velocities in the x and y directions are thresholded,in operation 406, with a minimum threshold (T_(m)) and a maximumthreshold (T_(M)), which can be either the same or different for the xand y directions. Thresholded velocities a and b for v_(x) and v_(y),respectively, may be computed as follows:

a=min(max(|v _(x) ^(t) |,T _(m)),T _(M)),

b=min(max(|v _(y) ^(t) |,T _(m)),T _(M))

As such, according to one embodiment of the present invention, the handtracking application 32 running on the image processing system 12calculates, in operation 408, an updated position (c_(t), r_(t)) of thecursor at time t:

c _(t)=min(max(c _(t-1) −v _(x) ^(t) *K*a,0),S _(w))

r _(t)=min(max(r _(t-1) −v _(y) ^(t) *K*b,0),S _(h))

where K is a sensitivity parameter controlling the sensitivity or gainof the movements on screen.

Therefore, by controlling the position of the cursor on the userinterface (e.g., as displayed on the display device 18) based on thevelocity of the gesture, where faster gestures of the same size resultin larger movements on screen. This enables a more comfortable andintuitive gesture based control, where small, fast gestures may be usedto move a cursor to distant portions of a display, while slow movementsprovide more precise control over the cursor. As such, differences inthe generated 2D positions (e.g., between subsequent positions) onscreen are non-linear with respect to the detected differences in the 3Dpositions (e.g., between subsequent positions) of the interactionmedium.

The thresholds and iterative updates based on velocity also free a userfrom being forced to remain within a fixed interaction zone in order tomaintain tracking, as will be described in more detail below.

In some embodiments of the present invention, the movement of the cursor(c_(t), r_(t)) may be smoothed by computing thresholded velocities a andb using average velocities computed over a time window instead of usinginstantaneous velocities v_(x) ^(t), and v_(y) ^(t). In some embodimentsof the present invention, acceleration along the x and y directions isused to compute the cursor position (c_(t), r_(t)).

In some embodiments, position dependent mapping may also be used to copewith detection systems that are characterized by different detectionperformance in different positions. For example, greater levels ofdetection noise may appear at the edges of the field of view of theacquisition system. Therefore, greater amounts of smoothing may beapplied when detecting gestures in those high noise regions. Inaddition, the remapping may be influenced by gesture detectionconfidence metrics supplied by the detection process.

Shifting Interaction Zone

Detecting gesturers across an entire field of view of an acquisitionsystem can be computationally expensive and can result in detectionerrors. However, as discussed above, limiting gestures to a fixedinteraction zone can result in frustrating user experiences.

According to one aspect of embodiments of the present invention, theinteraction zone may track interaction media (e.g., the body part or thecontrol device), thereby allowing use of the entire field of view of theacquisition device while maintaining the performance benefits of alimited interaction zone.

As discussed above, according to one embodiment, an initial interactionzone in which the detection is performed is defined based on thelocation of a wakeup event position P_(w)=[x_(w), y_(w), z_(w)]^(T), andmay be a cuboid of dimension Δ_(w)=[Δ_(w) ^(x), Δ_(w) ^(y), Δ_(w)^(z)]^(T). Within the interaction zone Δ_(w), another smaller internalcuboid may also be defined as an inner interaction zone Δ_(u), withdimension Δ_(u)=[Δ_(u) ^(x)<Δ_(w) ^(x),Δ_(u) ^(y)<Δ_(w) ^(y),Δ_(u)^(z)<Δ_(w) ^(z)]^(T) and also centered at P^(w), the inner interactionzone Δ_(u) being surrounded by the outer interaction zone Δ_(w).

FIGS. 5A and 5B are conceptual illustrations of inner and outerinteraction zones according to one embodiment of the present inventionand FIG. 6 is a flowchart illustrating a method for shifting aninteraction zone according to one embodiment of the present invention.

Referring to FIG. 6, in one embodiment, the hand tracking application 32running on the image processing system 12 sets the locations of theinner and outer interaction zones Δ_(w) and Δ_(u) in operation 610. FIG.5A illustrates an initial position of the inner and outer interactionzones with dotted lines. As discussed above, these may be initiallycentered on the wakeup event position P_(w). In operation 630, positioninformation regarding the position of the interaction medium 50 istracked as previously described to generate 3D position data P^(t).While the interaction medium 50 is tracked within both the outerinteraction zone Δ_(w) and the inner interaction zone Δ_(u), gesturedetection is only performed within the smaller internal cuboid Δ_(u). Ifa disengage event is detected in operation 650 (described in more detailbelow), then the process ends.

When the hand tracking application 32 detects of movement of theinteraction medium 50 out of the inner interaction zone Δ_(u) (e.g., ata boundary detection position P_(B) outside of Δ_(u) and within Δ_(w))in operation 670, then the outer interaction zone Δ_(w) and the innerinteraction zone Δ_(u) are shifted in operation 690 so as to include theboundary detection position P_(B) (e.g., by re-centering the interactionzone and the inner interaction zone at the boundary detection positionP_(B) and continuing to track movement based on the updated interactionzone). FIG. 5B illustrates one example in which the interaction medium50 has exited an upper part of the previsous inner interaction zoneΔ_(u) (depicted in dotted lines in FIG. 5B) at boundary detectionposition P_(B) and the subsequent shifting of the interaction zones tonew interaction zones (depicted in solid lines in FIG. 5B) centered atP_(B). After shifting the inner and outer interaction zones to newpositions, the process repeats with receiving position information andcontinuing to shift the interaction zones as necessary to track theposition of the interaction medium 50.

Some detection algorithms may use information about previous frameswithin the interaction zone in order to perform, for example, backgroundsubtraction or color modeling. In some embodiments, this data may bemade available during tracking by maintaining a buffer of recentlyreceived frames of data from the acquisition system 15 while trackingand by gathering information about the scene in the shifted interactionzone from the buffered frames.

In some embodiments of the present invention, hysteresis may be used toprovide smoother shifts of the interaction zone where, instead ofre-centering the interaction zone at the boundary detection positionP_(B), the interaction zone is re-centered at a position that is aconvex combination of the previous center of the interaction zone and ofthe boundary detection position P_(B). In still other embodiments of thepresent invention, the update is performed only using the coordinates ofthe detection that are not within the inner interaction cuboid Δ_(u).

In still another embodiment of the present invention, a detectiontechnique different from that used to detect gestures may be used todetect movement that triggers an update of the interaction zone. Forexample, a coarse detection technique may be applied to a largerdetection region than the interaction zone. In one embodiment, a fingermay be tracked as the interaction medium and the gesture detectionmethod may track the fingertip within a relatively small interactionzone (e.g., using template based tracking or edge detection strategies)while movement out of the interaction zone may be detected by coarselydetecting the position of the hand in the entire field of view of theacquisition system (or within a larger region that includes theinteraction zone). The coarse position of the hand can then be used tore-center the interaction zone while maintaining more refined detectionof the position of the finger within the interaction zone.

Disengagement

In the case of a fixed interaction zone, a user could disengage (e.g.,cause the system to stop tracking gestures) by moving the interactionmedium (e.g., body part or stylus) out of the interaction zone. However,such a disengagement gesture would not work easily in embodiments of thepresent invention in which the interaction zone dynamically tracks orfollows the movement of the interaction medium throughout the field ofview of the acquisition system.

Therefore, in some embodiments of the present invention, the systemdetects a disengagement event in operation 650 to cause the system tostop tracking gestures made by the user. FIGS. 7A and 7B are flowchartsillustrating methods for detecting a disengagement event according tosome embodiments of the present invention.

Referring to FIG. 7A, in one embodiment, the system tracks a particularshape (such as a pointing finger or open hand) when performing cursorcontrol. In such embodiments, it is possible to perform a negativedetection procedure (using hypothesis testing) to stop tracking when thetracked shape is no longer detected (e.g., when the user stops pointinga single finger or when the user closes his or her open hand). Forexample, at each detection event, the detected shape is tested forcompatibility with the target configuration (e.g., a confidence withwhich the detected shape matches the shape of a hand with a pointingfinger) in operation 652. The confidence levels of the compatibility arecompared to a threshold value in operation 654. If all of the confidencelevels are below the threshold value (e.g., the detected shape does notresemble any of the known ways that the target configuration couldappear), then tracking stops in operation 656. However, if the detectedshape does match the target configuration, then tracking continues inoperation 658 and the process continues (e.g., by proceeding withdetermining if the interaction zone needs to be shifted in operation 670of FIG. 6).

In some embodiments, in the case of an open hand and an acquisitionsystem made by standard video cameras, it is possible to apply computervision and machine learning techniques such as the analysis ofHistograms of Gradients (HoG) to provide such a hypothesis testinganalysis.

FIG. 7B illustrates another embodiment in which a specific disengagementgesture is be used to end tracking For example, in the case of handgesture based interactions, it may be natural for a user to lower his orher hand when the user is done with interacting with the system. In sucha case, a downward motion of a particular length and over a particulartime frame may be used to provide disengagement. In operation 652′ thedata from the acquisition system is compared with information regardinga disengagement configuration and, if the disengagement gesture isdetected in 654′, then the system stops tracking in operation 658′. Insome embodiments, dynamic time warping techniques are applied to detectdisengagement gestures.

FIG. 7C illustrates another embodiment in which a disengagement event isdetected when the image processing system 12 receives information inoperation 652″ that the finger moves fast enough to transition from theinner interaction zone Δ_(u) into the outer interaction zone Δ_(w) andsubsequently out of the outer interaction zone Δ_(w), as detected inoperation 654″, within a disengagement time period, such as before theinteraction zone update time (e.g., 1 frame or ˜33 ms, assuming that theacquisition system is operating at 30 frames per second).

Boundary Hysteresis

One aspect of embodiments of the present invention allows a user toreposition the interaction zone without changing the position of thecursor. For example, a user may make a wake up gesture at head heightand then find that the interaction zone is uncomfortably high. Ratherthan disengage and reengage the gesture detection system, one embodimentof the present invention allows a user to “drag” the interaction zone toa more comfortable location. For example, if the interaction zone is toohigh, the user may move his or her hand in a downward direction.Initially, the cursor on screen would also move downward, but wouldeventually stop moving at the bottom edge of the user interface becauseits coordinates always lie within [0 . . . S_(w)] and [0 . . . S_(h)].For example, the cursor would be “pinned” or “clamped” at the edge ofthe screen in the same way that a mouse cursor would typically pinned atthe edge of a screen even if the user continued moving the physicalmouse.

Nevertheless, the user's hand would continue moving downward and thegesture detection system would continue tracking the hand and continueupdating the position of the inner and outer interaction zones Δ_(u) andΔ_(w). Once the user had lowered his or her hand to comfortable height,raising the hand would immediately cause cursor movement in the upwarddirection (e.g., in the same way that a mouse cursor would move upwardonce a user moved in the opposite direction). As such, the distancemoved by the hand to bring the cursor down to the bottom of the displayis longer than the distance moved by the hand to bring the cursor backup to the center of the display, and this difference may be referred toherein as “boundary hysteresis.”

FIG. 8 is a flowchart illustrating a method for repositioning aninteraction zone according to one embodiment of the present invention.In operation 802, inner and outer interaction zone locations may be setand tracked as described above. In operation 804, the movement of aninteraction medium 50 along a first direction (e.g., a downwarddirection) may then be tracked past the edge of the interaction zonesand a cursor displayed on the user interface may be moved along thefirst direction in response to the tracked movement. The interactionzones are shifted along the first direction in operation 806 and thecursor stops moving (e.g., because the cursor has reached a boundarysuch as the edge of the screen). In operation 808, the image processingsystem 12 begins tracking movement along a second direction opposite thefirst direction (e.g., movement in an upward direction) and concurrentlybegins moving the cursor in the second direction.

As such, aspects of embodiments of the present invention allow a user toreposition an interaction zone in a manner that parallels therepositioning of a mouse on a mousepad or the repositioning of a fingeron a touchpad.

Therefore, aspects of embodiments of the present invention allownon-linear remapping of gestures to a display device as well as trackingwithin an entire field of view of the acquisition system whilemaintaining benefits of a relatively small interaction zone. Theseaspects provide a more familiar interface for users accustomed to usingstandard computer mice and trackpads. The non-linear mapping allowsusers to have more precise control by slowing down the velocity of theirinteractions while also allowing more agile interactions by making morerapid movements. The nonlinear remapping may also include parametersthat can be adjusted by a user to control the sensitivity and responsecharacteristics of the system.

In addition, aspects of embodiments of the present invention allowdecoupling of the 2D position of the cursor on the screen from theabsolute 3D position of the interaction medium in space, therebyallowing remapping based on relative movements in the 3D space. Thisalso allows a user to control the system from very different regionswithin the field of view of the acquisition system during an interactionsession without actively disengaging and reengaging the detectionsystem.

While the present invention has been described in connection withcertain exemplary embodiments, it is to be understood that the inventionis not limited to the disclosed embodiments, but, on the contrary, isintended to cover various modifications and equivalent arrangementsincluded within the spirit and scope of the appended claims, andequivalents thereof. For example, the features and aspects describedherein may be implemented independently, cooperatively or alternativelywithout deviating from the spirit of the disclosure.

What is claimed is:
 1. A real-time gesture based interactive systemcomprising: a processor; an acquisition system configured to capture asequence of frames of data for constructing a depth map of a field ofview of the acquisition system; memory storing an interaction mediumtracking application, the interaction medium tracking applicationconfiguring the processor to: obtain the data from the acquisitionsystem; compare successive frames of the frames of data for portionsthat change from one frame to the next; determine whether any of theportions that changed are part of an interaction medium detected in thesequence of frames of data; define an inner 3D interaction zone relativeto an initial position of the part of the interaction medium detected inthe sequence of frames of data, where the inner 3D interaction zonecorresponds to a bounded region that is less than the frame of data andthat contains the part of the interaction medium detected in thesequence of frames of data; track a movement of the interaction mediumin the data to generate a plurality of 3D positions of the interactionmedium; detect movement of the interaction medium from inside to outsidethe inner 3D interaction zone at a boundary 3D position; shift the inner3D interaction zone relative to the boundary 3D position; compute aplurality of 2D positions based on the 3D positions; and supply the 2Dpositions to control an application.
 2. The system of claim 1, whereinthe acquisition system comprises at least one of: a plurality of camerasin a stereo arrangement having overlapping fields of view; an infraredcamera; a color visible light camera; and an illumination sourceconfigured to generate at least one of visible light, infrared light,ultrasonic waves, and electromagnetic signals.
 3. The system of claim 1,wherein the memory further stores instructions of the interaction mediumtracking application to configure the processor to: compute a 3Dvelocity of the interaction medium within the inner 3D interaction zonebased on the 3D positions; and compute two-dimensional movement datacorresponding to the 3D positions and the 3D velocity, differences inthe two-dimensional movement data being non-linear with respect todifferences in the 3D positions.
 4. The system of claim 1, wherein theinteraction medium is a portion of a human body.
 5. The system of claim1, wherein the memory further stores instructions of the interactionmedium tracking application to configure the processor to detect themovement of the interaction medium from the inside to the outside of theinner 3D interaction zone based on the 3D positions.
 6. The system ofclaim 1, wherein the memory further stores instructions of theinteraction medium tracking application to configure the processor todetect the movement of the interaction medium from the inside to theoutside of the inner 3D interaction zone by detecting a coarse movementof the interaction medium within an entire portion of the field of viewof the acquisition system.
 7. The system of claim 1, wherein the memoryfurther stores instructions of the interaction medium trackingapplication to configure the processor to: detect a disengagement event;and stop tracking the movement of the interaction medium in response tothe disengagement event.
 8. The system of claim 7, wherein the memoryfurther stores instructions of the interaction medium trackingapplication to configure the processor to detect the disengagement eventby: comparing the interaction medium detected in the frames of data to atarget configuration to generate a compatibility confidence level; anddetecting the disengagement event when the confidence level is below athreshold level.
 9. The system of claim 7, wherein the memory furtherstores instructions of the interaction medium tracking application toconfigure the processor to detect the disengagement event by: comparingthe interaction medium detected in the frames of data to a disengagementconfiguration; and detecting the disengagement event when thedisengagement configuration is detected.
 10. The system of claim 1,wherein the memory further stores instructions of the interaction mediumtracking application to configure the processor to: track the movementof the interaction medium along a first direction toward a boundary ofthe inner interaction zone and concurrently update the plurality of 2Dpositions in accordance with the movement along the first direction;shift the inner interaction zone along the first direction andconcurrently stop updating the plurality of 2D positions in accordancewith movement along the first direction; and track the movement of theinteraction medium along a second direction opposite the first directionand concurrently start updating the plurality of 2D positions inaccordance with the movement along the second direction.
 11. A real-timegesture based interactive system comprising: a processor; an acquisitionsystem configured to capture a sequence of frames of data forconstructing a depth map of a field of view of the acquisition system;memory storing an interaction medium tracking application, theinteraction medium tracking application configuring the processor to:obtain the data from the acquisition system; compare successive framesof the frames of data for portions that change from one frame to thenext; determine whether any of the portions that changed are part of aninteraction medium detected in the sequence of frames of data; define aninner 3D interaction zone relative to an initial position of the part ofthe interaction medium detected in the sequence of frames of data, wherethe inner 3D interaction zone corresponds to a bounded region that isless than the frame of data and that contains the part of theinteraction medium detected in the sequence of frames of data; track amovement of the interaction medium in the data to generate a pluralityof 3D positions of the interaction medium; compute a 3D velocity of theinteraction medium within the inner 3D interaction zone based on the 3Dpositions; compute a plurality of 2D positions based on the 3D positionsand the 3D velocity, differences in the 2D positions being non-linearwith respect to differences in the 3D positions; and supply the 2Dpositions to control an application.
 12. The system of claim 11, whereinthe 3D velocity comprises a horizontal component v_(x) and a verticalcomponent v_(y) and wherein the 3D velocity is thresholded by a minimumthreshold T_(m) and a maximum threshold T_(M) to compute thresholded 3Dvelocities a and b, where:a=min(max(|v _(x) ^(t) |,T _(m)),T _(M)),b=min(max(|v _(y) ^(t) |,T _(m)),T _(M)).
 13. The system of claim 12,wherein the real-time gesture based interactive system further comprisesa display device having S_(w) columns and S_(h) rows, wherein theplurality of 2D positions comprises column data c and row data rcorresponding to coordinates of the display device, and wherein the 3Dvelocity and the thresholded 3D velocities are mapped onto the columndata c and the row data r, where:c _(t)=min(max(c _(t-1) −v _(x) ^(t) *K*a,0),S _(w))r _(t)=min(max(r _(t-1) −v _(y) ^(t) *K*b,0),S _(h)). and where K is asensitivity parameter.
 14. A method for operating a real-time gesturebased interactive system, the method comprising: obtaining a sequence offrames of data from an acquisition system configured to capture data forconstructing a depth map of a field of view of the acquisition system;comparing, by a processor, successive frames the frames of data forportions that change from one frame to the next; determining, by theprocessor, whether any of the portions that changed are part of aninteraction medium detected in the sequence of frames of data; defining,by the processor, an inner 3D interaction zone relative to an initialposition of the part of the interaction medium detected in the sequenceof frames of data, where the inner 3D interaction zone corresponds to abounded region that is less than the frame of data and that contains thepart of the interaction medium detected in the sequence of frames ofdata; tracking, by the processor, a movement of the interaction mediumin the data to generate a plurality of 3D positions of the interactionmedium; detecting, by the processor, movement of the interaction mediumfrom inside to outside the inner 3D interaction zone at a boundary 3Dposition; shifting, by the processor, the inner 3D interaction zonerelative to the boundary 3D position; computing, by the processor, aplurality of 2D positions based on the 3D positions; and supplying the2D positions to control an application.
 15. The method of claim 14,wherein the acquisition system comprises at least one of: a plurality ofcameras in a stereo arrangement having overlapping fields of view; aninfrared camera; a color visible light camera; and an illuminationsource configured to generate at least one of visible light, infraredlight, ultrasonic waves, and electromagnetic signals.
 16. The method ofclaim 14, further comprising: computing a 3D velocity of the interactionmedium within the inner 3D interaction zone based on the 3D positions;and computing two-dimensional movement data corresponding to the 3Dpositions and the 3D velocity, differences in the two-dimensionalmovement data being non-linear with respect to differences in the 3Dpositions.
 17. The method of claim 14, wherein the interaction medium isa portion of a human body.
 18. The method of claim 14, furthercomprising detecting the movement of the interaction medium from theinside to the outside of the inner 3D interaction zone based on the 3Dpositions.
 19. The method of claim 14, further comprising detecting themovement of the interaction medium from the inside to the outside of theinner 3D interaction zone by detecting a coarse movement of theinteraction medium within an entire portion of the field of view of theacquisition system.
 20. The method of claim 14, further comprising:detecting a disengagement event; and stopping tracking the movement ofthe interaction medium in response to the disengagement event.
 21. Themethod of claim 20, wherein the detecting the disengagement eventcomprises: comparing the interaction medium detected in the frames ofdata to a target configuration to generate a compatibility confidencelevel; and detecting the disengagement event when the confidence levelis below a threshold level.
 22. The method of claim 20, wherein thedetecting the disengagement event comprises: comparing the interactionmedium detected in the frames of data to a disengagement configuration;and detecting the disengagement event when the disengagementconfiguration is detected.
 23. The method of claim 14, furthercomprising: tracking the movement of the interaction medium along afirst direction toward a boundary of the inner interaction zone andconcurrently update the plurality of 2D positions in accordance with themovement along the first direction; shifting the inner interaction zonealong the first direction and concurrently stop updating the pluralityof 2D positions in accordance with movement along the first direction;and tracking the movement of the interaction medium along a seconddirection opposite the first direction and concurrently start updatingthe plurality of 2D positions in accordance with the movement along thesecond direction.
 24. A method for tracking a gesture comprising:obtaining a sequence of frames of data from an acquisition systemconfigured to capture data for constructing a depth map of a field ofview of the acquisition system; comparing, by a processor, successiveframes of the frames of data for portions that change from one frame tothe next; determining, by the processor, whether any of the portionsthat changed are part of an interaction medium detected in the sequenceof frames of data; defining, by the processor, an inner 3D interactionzone relative to an initial position of the part of the interactionmedium detected in the sequence of frames of data, where the inner 3Dinteraction zone corresponds to a bounded region that is less than theframe of data and that contains the part of the interaction mediumdetected in the sequence of frames of data; tracking, by the processor,a movement of the interaction medium in the data to generate a pluralityof 3D positions of the interaction medium; computing, by the processor,a 3D velocity of the interaction medium within the inner 3D interactionzone based on the 3D positions; computing, by the processor, a pluralityof 2D positions based on the 3D positions and the 3D velocity,differences in the 2D positions being non-linear with respect todifferences in the 3D positions; and supplying, by the processor, the 2Dpositions to control an application.
 25. The method of claim 24, whereinthe 3D velocity comprises a horizontal component v_(x) and a verticalcomponent v_(y) and wherein the 3D velocity is thresholded by a minimumthreshold T_(m) and a maximum threshold T_(M) to compute thresholded 3Dvelocities a and b, where:a=min(max(|v _(x) ^(t) |,T _(m)),T _(M)),b=min(max(|v _(y) ^(t) |,T _(m)),T _(M)).
 26. The method of claim 25,wherein the application includes a user interface displayed on a displaydevice having S_(w) columns and S_(h) rows, wherein the plurality of 2Dpositions comprises column data c and row data r corresponding tocoordinates of the display device, and wherein the 3D velocity and thethresholded 3D velocities are mapped onto the column data c and the rowdata r, where:c _(t)=min(max(c _(t-1) −v _(x) ^(t) *K*a,0),S _(w))r _(t)=min(max(r _(t-1) −v _(y) ^(t) *K*b,0),S _(h)). and where K is asensitivity parameter.